System and method for electronic correction of optical anomalies

ABSTRACT

An electronic correction system and method for correcting optical anomalies, namely distortions, color non-convergence (excluding axial chromatic aberration) and luminance (or chrominance) non-uniformity. Each effect is modeled as a transformation in either spatial (positional) space or color space. Representing the effects as transformations of digital pixel data, allows the different anomalies to be resolved within a common framework, namely that of image ‘warping’. The anomaly, having been expressed as a pixel transformation, is then eliminated by electronically applying the inverse transformation. This process is equivalent to digitally manipulating or warping the image in position and/or color space and accordingly this can be achieved using commercially known warping circuits. In addition, the transformation can also contain a component to additionally perform any application specific image warping (e.g. scaling and geometric transformations). Finally, by storing a sequence of transformations, adaptive anomaly correction and dynamic effects can be achieved.

[0001] This application claims the benefit under 35 U.S.C. 119(e) ofU.S. Provisional Application No. 60/387,596, filed Jun. 12, 2002.

FIELD OF THE INVENTION

[0002] This invention relates to electronic correction of opticalanomalies and more particularly to electronic correction of distortion,lateral chromatic aberration, luminance non-uniformity,and chrominancenon-uniformity, which can be combined with a general image transform(e.g. scaling and geometric transform).

BACKGROUND OF THE INVENTION

[0003] Conventional image capture or display devices are prone tovarious forms of optical anomalies. These anomalies are inherent to thenon-ideal behavior of the various optical elements and to accuracy andtolerance of assembly. Various components (sensors, displays, lens,prisms, mirrors, light source), optical or otherwise, and theirorientation may introduce their own specific optical anomalies, such asdistortion, tilt, lateral chromatic aberration, luminance or chrominancenon-uniformity. The term optical aberration is generally used to referto any effect that leads to an non-ideal image formation.

[0004] Optical aberrations include diffraction effects (due to the wavenature of light), chromatic aberrations (caused by optical dispersion,or the differences in refraction in different wavelengths of light), andmonochromatic aberrations (of which spherical aberration, coma, andastigmatism are concerned with failures of a point object to form apoint image, and field curvature and distortion are concerned with thefailure of finite objects perpendicular to the principal axis to form awell-focused plane image). One can also loosely group opticalaberrations into two types, ones that affect image quality and ones thataffect image shape. The former types degrade the sharpness of an image,that is, the image appears blurred, and/or out of focus, and/or hascolor fringes. Aberrations in this category include sphericalaberrations, astigmatism, coma, field of curvature and axial chromaticaberration. The latter types of aberrations affect the shape of theimage that in part may be induced by the former aberration or by thecorrection and optimization of the former aberration. In this case,points in the object plane are shifted or distorted in comparison withan ideal mapping in the image plane. In an ideal mapping an object inthe image plane will appear as it does in the object plane, withpossibly a uniform scale change. For example, an image may appear curvednear the edges or appear rotated. Aberrations in this second categoryinclude distortions (e.g. pincushion/barrel effects) and lateralchromatic aberrations.

[0005] With the exception of chromatic aberrations, all other opticalaberrations are present in monochromatic (i.e. single color) light.Chromatic aberration appears when dealing with polychromatic light (manycolors). In short, the index of refraction is wavelength dependent,which means that the red, green and blue components bend differently atan optical interface. This leads to axial (longitudinal) and/or lateralchromatic aberration effects. In axial chromatic aberration, the threecomponents are brought to focus on different planes in the image space,which gives a color blurring effect. In other words, axial chromaticaberration arises due to the focal length varying with wavelength(color). In lateral chromatic aberration, color components from a singlepoint are brought to focus to different points on the same image plane.This has the effect of magnifying the three colors differently and canbe visually seen as ‘color fringing’. Thus lateral chromatic aberrationcan be seen as an effect due to magnification varying with wavelength.The three colors can also mismatch due to non-optical effects. In athree-color display system, if the displays are not correctly aligned,color defects will be seen. The term color non-convergence is used torefer to color mismatch effects, whether optical (as in chromaticaberrations) or not. Further discussion on optical aberrations can befound in conventional optics textbooks, such as Robert Guenther's ModernOptics, published by John Wiley & Sons, 1990, hereby incorporated byreference.

[0006] Another important optical anomaly in conventional capture/displaydevices is luminance non-uniformity. Luminance non-uniformity leads tovarying brightness across an image. Common causes include a varying (inbrightness) light source, varying optical path across the image plane,non-uniform sensor response and irregularities in panels (e.g. LCD,LCOS, etc.). Both large-scale and small-scale non-uniformities can bepresent. In a three-color system, brightness variation can be differentfor each color, leading to chrominance non-uniformity.

[0007] There have been a number of prior art attempts to correctaberrations that affect the shape of the image, without introducingblur, namely correction of distortion, lateral chromatic aberration andluminance or chrominance (brightness) non-uniformity. Generally, suchprior art attempts are geared towards one specific type of anomaly.

[0008] For example, lateral chromatic aberration is commonly corrected(or minimized) using special optical elements, often consisting ofprism/lens combinations and/or special material coatings such as thatdisclosed in U.S. Pat. No. 4,943,155 to Cross, U.S. Pat. No. 5,086,338to Usui, U.S. Pat. No. 5,499,139 to Chen et al., U.S. Pat. No. 6,023,375to Kreitzer, U.S. Pat. No. 6,111,701 to Brown, U.S. Pat. No. 6,144, 498to Bryars et al., and U.S. Pat. No. 6,172,815 to Hashizume et al.However, the physical implementation of the solutions disclosed in thesereferences are expensive and bulky. Further, the specialized nature ofthese designs necessarily restrict them to specific types ofapplications. Typically, these methods are aimed at display/projectionsystems or head up/mounted display systems.

[0009] A number of electronic solutions have also been presented such asthose disclosed in U.S. Pat. No. 5,838,396 to Shiota et al., U.S. Pat.No. 5,870,505 to Munib et al., U.S. Pat. No. 6,288,756 to Shiota et al.,U.S. Pat. No. 5,200,815 to Tsujihara et al., U.S. Pat. No. 5,369,450 toHaseltine et al, 5,889,625 to Chen et al., and U.S. Pat. No. 6,323,934to Enomoto et al. All of these approaches rely on some type of image“warping”. A discussion of image warping can be found in GeorgeWolberg,Digial Image Warping, IEEE Computer Society Press, 1988, herebyincorporated by reference.

[0010] The warping data (i.e. data which describes how the image is tobe transformed) may be used to adjust the digital image (e.g. as in U.S.Pat. No. 5,369,450 to Haseltine et al. and U.S. Pat. No. 5,889,625 toChen et al.) or to adjust the operation of the electronics thatdisplay/project the image (e.g. as in U.S. Pat. No. 5,200,815). Theseelectronic solutions concentrate on specific anomalies, such asluminance correction (e.g. U.S. Pat No. 5,838,396 to Shiota et al., U.S.Pat. No. 5,870,505 to Woeber et al, U.S. Pat. No. 6,288,756 to Shiota etal.), distortion and chromatic aberration (e.g. U.S. Pat. No. 5,200,815to Tsujihara et al., U.S. Pat. No. 5,369,450 to Haseltine et al, U.S.Pat. No. 5,889,625 to Chen et al., and U.S. Pat. No. 6,323,934 toEnomoto et al.) or specific types of systems, such as head-mounteddisplays. Those solutions that do correct for all three anomalies (e.g.U.S. Pat. No. 6,323,934 to Enomoto et al.) are not real-time in nature.Other limitations of prior art electronic solutions are that they do notallow for application specific “correction” (i.e. correction which doesnot correspond to an optical anomaly correction) and/or they do notprovide for dynamic anomaly correction. For example, it can be desirablein certain video applications to combine anomaly correction (brightnessnon-uniformity as well as pincushion distortion) with a keystonecorrection (caused by off-axis projection) and curvature correction forplanar/curved surfaces.

SUMMARY OF THE INVENTION

[0011] The invention provides in one aspect, electronic correctionmethod for correcting a plurality of optical anomalies associated withthe capture and display of an optical image processed through opticalcapture and display components having a particular geometry, bycompensation of the digital image pixel data associated with the opticalimage, said method comprising:

[0012] (a) identifying and representing the optical anomalies associatedwith the physical and geometrical characteristics of the capture anddisplay optical components as an optical anomaly grid dataset;

[0013] (b) identifying and representing the ideal behavior of thecapture and display optical components as an ideal grid dataset;

[0014] (c) comparing the optical anomaly grid dataset with the idealgrid dataset and determining an anomaly correcting transformationdataset by performing an inverse spatial transform from the ideal griddataset to the anomaly transform such that functional composition of theanomaly correcting transformation with the optical anomaly grid datasetreduces to the ideal grid dataset;

[0015] (d) applying the anomaly correcting transformation dataset to theimage pixel data to produce corrected image pixel data which when viewedis free of the optical anomaly.

[0016] The invention provides in another aspect, an electroniccorrection system for correcting a plurality of optical anomaliesassociated with the capture and display of an optical image processedthrough optical capture and display components having a particulargeometry, by compensation of the digital image pixel data associatedwith the optical image, said system comprising an image processor for:

[0017] (a) identifying and representing the optical anomalies of thephysical and geometrical characteristics of the capture and displayoptical components as an optical anomaly grid dataset;

[0018] (b) identifying and representing the ideal behavior of the imagedata processing chain as an ideal grid dataset;

[0019] (c) comparing the optical anomaly grid dataset with the idealgrid dataset and determining an anomaly correcting transformationdataset by performing an inverse spatial transform from the ideal griddataset to the anomaly transform such that functional composition of theanomaly correcting transformation with the optical anomaly grid datasetreduces to the ideal grid dataset;

[0020] (d) applying the anomaly correcting transformation dataset to theimage pixel data to produce corrected image pixel data which when viewedis free of the optical anomaly.

[0021] The invention provides in another aspect, a computer-readablemedium having computer-readable code embodied therein for correcting aplurality of optical anomalies associated with the capture and displayof an optical image processed through optical capture and displaycomponents having a particular geometry, by compensation of the digitalimage pixel data associated with the optical image, by:

[0022] (a) identifying and representing the optical anomalies of thephysical and geometrical characteristics of the capture and displayoptical components as an optical anomaly grid dataset;

[0023] (b) identifying and representing the ideal behavior of the imagedata processing chain as an ideal grid dataset;

[0024] (c) comparing the optical anomaly grid dataset with the idealgrid dataset and determining an anomaly correcting transformationdataset by performing an inverse spatial transform from the ideal griddataset to the anomaly transform such that functional composition of theanomaly correcting transformation with the optical anomaly grid datasetreduces to the ideal grid dataset;

[0025] (d) applying the anomaly correcting transformation dataset to theimage pixel data to produce corrected image pixel data which when viewedis free of the optical anomaly.

[0026] Further aspects and advantages of the invention will appear fromthe following description taken together with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the accompanying drawings:

[0028]FIG. 1 is a schematic diagram illustrating a conventional priorart integrated capture and display device and its corresponding imageprocessing chain;

[0029]FIG. 2 is a schematic diagram illustrating a detailed processingchain and the various transformations that represent the optical defectswithin the prior art integrated capture and display device of FIG. 1;

[0030]FIG. 3 is a schematic diagram that illustrates an ideal processingchain in the absence of any optical defects;

[0031]FIG. 4 is a schematic diagram of an example of an electroniccorrection system of the present invention;

[0032]FIG. 5 is a schematic diagram of the correction module of theelectronic correction system of FIG. 4 where the correction module hasbeen adapted to correct for distortion and chrominance non-uniformity;

[0033]FIG. 6 is a schematic diagram of another example of an electroniccorrection system of the present invention that executes a geometrictransform;

[0034]FIG. 7 is a schematic diagram of the correction module of theelectronic correction system of FIG. 6 where the correction module hasbeen adapted to implement an application specific geometric transform;

[0035]FIG. 8 is a schematic diagram of another example of an electroniccorrection system of the present invention that corrects opticalanomalies, executes geometric transforms, and achieves dynamiccorrection; and

[0036]FIG. 9 is a flowchart illustrating the main steps of theelectronic correction method of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0037]FIG. 1 illustrates the main elements of a conventional imagecapture/display device 10 and the optical anomalies present within thecorresponding image processing chain that is represented as image (i.e.pixel level) transformations. As shown, an object O in the object planeis mapped, via the image processing chain, onto an object O′ in theimage plane. Capture/display device 10 contains various elements,including optical elements e.g. lenses, prisms, mirrors, etc.) andnon-optical elements (e.g. electronic sensors and display panels) 12 and14 some or all of which may introduce an optical anomaly into the imageprocessing chain as shown. Capture/display device 10 also includesmemory (i.e. storage) 16 and 18 as shown.

[0038] Each optical anomaly acts in the spatial space (affecting theshape of the imaged object) and/or in the color space (affecting thebrightness/color of the imaged object). These effects can be representedas transformations or mappings of (a) pixel positions and (b) pixelcolors, that are applied somewhere between the memory plane (space) 16and 18 and the corresponding object/image plane. Accordingly, thesetransformations define the optical anomalies. It should be understoodthat the current representation does not consider differential focusanomalies as differential focus problems cannot be correctedelectronically.

[0039]FIG. 1 shows the processing chain where the maps representing thecollective effect of all the anomalies on the capture side (defined bysubscript c) by ƒ_(c) and on the display side (defined by subscript d)by ƒ_(d). The optical anomalies present within capture/display device 10are represented by these maps. The map I associated with storage at thememory plane represents the identity map, which indicates that is thereis no change in the image as the memory planes 16 and 18 do notintroduce any anomalies. It should be noted that the term opticalanomaly is used to refer to any effect that leads to an non-ideal imageformation, in particular this includes aberrations and luminancenon-uniformity.

[0040]FIG. 2 illustrates how the map ƒ_(c) can be further split intof_(c) ^(s) and f_(c) ^(c), where f_(c) ^(s) describes spatial anomalies(superscript s) and ƒ_(c) ^(c) describes color anomalies (superscriptc). Since each color component may transform differently, ƒ_(c) ^(s) andƒ_(c) ^(c) should be replaced byf_(cr)^(s), f_(cg)^(s), f_(cb)^(s)  and  f_(cr)^(c), f_(cg)^(c), f_(cb)^(c),

[0041] for the red, green and blue components (second subscripts r,g,b).The notation ƒ_(c) ^(s) and ƒ_(c) ^(c) will be used when referring toall three component functions together. Similarly, the spatial/colormaps representing the anomalies on the display side are denoted by ƒ_(d)^(s) and ƒ_(d) ^(c) with their color componentsf_(dr)^(s), f_(dg)^(s), f_(db)^(s), f_(dr)^(c), f_(dg)^(c)

[0042] and f_(db)^(c).

[0043] It should be noted that the specifics (size, type, etc.) of thememory planes are system dependent and that the memory planes shownrepresent conceptual domains for the various maps rather than actualhardware implementations of storage. In particular, the memory plane onthe display side is shown in dotted outline to indicate that it need notbe present in a specific device, Instead, a captured image may bedirectly displayed without being stored. If additional processing isconducted in the display part of capture/display device 10 (e.g. framerate conversion) then some form of physical memory will be required.Furthermore, the memory plane on the capture side does need not be fullframe.

[0044] An object O in the object plane is defined by its 2D positionvector

(for a point) and its RGB value (O_(r),O_(g),O_(b)) In its digital form,the point object is approximated by a pixel. Due to chromaticaberration, the color components of this object will be mapped todifferent points in the image plane. In general, the single point willbe mapped to three points as the three colors are brought to focus todifferent points (on the same plane). Thus, the image of the object inthe image plane will be described by three position vectors (

′_(r),

′_(g),

′_(b)) and the corresponding color values (O′_(r), O′_(g), O′_(b)) It isalso useful to introduce the notation (

_(r),

_(g),

_(b)) for the spatial coordinates of the three colors in the objectspace, with the understanding that for a given point we have

=

_(r)=

_(g)=

_(b).

[0045] Combining everything, the processing of a point object O (orpixel) through capture/display device 10 in the presence of opticalanomalies can be expressed by the following mappings (all spatial mapsare represented as vector functions, although to simplify notation thevector arrow has been omitted): $\begin{matrix}\left. \left( {\overset{\rho}{x},O_{r},O_{g},O_{b}} \right)\rightarrow\left( {\overset{\rho}{x_{r}^{\prime}},\overset{\rho}{x_{g}^{\prime}},\overset{\rho}{x_{b}^{\prime}},O_{r}^{\prime},O_{g}^{\prime},O_{b}^{\prime}} \right) \right. & (1) \\{\overset{\rho}{x_{r}^{\prime}} = {f_{dr}^{s}\left( {f_{cr}^{s}\left( \overset{\rho}{x} \right)} \right)}} & (2) \\{\overset{\rho}{x_{g}^{\prime}} = {f_{dg}^{s}\left( {f_{cg}^{s}\left( \overset{\rho}{x} \right)} \right)}} & (3) \\{\overset{\rho}{x_{b}^{\prime}} = {f_{db}^{s}\left( {f_{cb}^{s}\left( \overset{\rho}{x} \right)} \right)}} & (4) \\{O_{r}^{\prime} = {f_{dr}^{c}\left( {f_{cr}^{c}\left( O_{r} \right)} \right)}} & (5) \\{O_{g}^{\prime} = {f_{dg}^{c}\left( {f_{cg}^{c}\left( O_{g} \right)} \right)}} & (6) \\{O_{b}^{\prime} = {{f_{db}^{c}\left( {f_{cb}^{c}\left( O_{b} \right)} \right)}.}} & (7)\end{matrix}$

[0046] The corresponding image processing chain is shown in FIG. 2. Thenotation q(z), for a mapping q and variable z, means that q acts tomodify z only, changing it to a new value of the same type (i.e. a newposition value or a new color value). The mapping q can also depend onother variables, however in order to simplify the notation, these arenot shown. For example, z′=q(z,w) is written as z′=q(z). In general, allspatial mappings depend only on spatial variables, whereas the colormappings can depend on both spatial and color variables, as will befurther described.

[0047]FIG. 3 illustrates the ideal behaviour of capture/display device10. To correct the various optical anomalies, we also need to describethe ‘ideal’ capture/display device in the absence of any anomalies. Thecorrecting maps will try to restore this ideal behavior. In the absenceof any lateral chromatic aberration we will have the following relation:$\begin{matrix}{{\overset{\rho}{x_{r}^{\prime}} = {\overset{\rho}{x_{g}^{\prime}} = {\overset{\rho}{x_{b}^{\prime}} \equiv \overset{\rho}{x^{\prime}}}}},} & (8)\end{matrix}$

[0048] where $\overset{\rho}{x^{\prime}}$

[0049] represents the common vector for all three colors. Additionally,in the absence of any distortion one expects the following relation:$\begin{matrix}{{\overset{\rho}{x^{\prime}} = {k\overset{\rho}{x}}},} & (9)\end{matrix}$

[0050] where capture/display device 10 only linearly scales objects inthe object plane. The value “k” represents a spatial scaling constant.Thus zooming operations are not considered as distortions (when suchoperations are identical for each color). Lastly, for uniformluminance/chrominance one expects the following relation:

O′_(r)=sO_(r)+m  (10)

O′_(g)=sO_(g)+m  (11)

O′_(b)=sO_(b)+m,  (12)

[0051] where the same change in each color component occurs all acrossthe image (independent of

). The “s” and “m” values represent the uniform luminance/chrominancegain and offset values applied, respectively. The processing of a pointobject O (or pixel) through capture/display device 10 in the absence ofany optical anomalies can be expressed by the following mappings (usingh for the ideal mappings): $\begin{matrix}{\left( {\overset{\rho}{x},O_{r},O_{g},O_{b}} \right)->\left( {{\overset{\rho}{x}}^{\prime},O_{r}^{\prime},O_{g}^{\prime},O_{b}^{\prime}} \right)} & (13) \\{{\overset{\rho}{x}}^{\prime} = {h_{d}^{s}\left( {h_{c}^{s}\left( \overset{\rho}{x} \right)} \right)}} & (14) \\{O_{r}^{\prime} = {h_{d}^{c}\left( {h_{c}^{c}\left( O_{r} \right)} \right)}} & (15) \\{O_{g}^{\prime} = {h_{d}^{c}\left( {h_{c}^{c}\left( O_{g} \right)} \right)}} & (16) \\{{O_{b}^{\prime} = {h_{d}^{c}\left( {h_{c}^{c}\left( O_{b} \right)} \right)}},} & (17)\end{matrix}$

[0052] where h_(c) ^(s), h_(c) ^(c), h_(d) ^(s) and h_(d) ^(c) areconstrained by equations (9)-(12), and are limiting forms forf_(ci)^(s), f_(ci)^(c), f_(di)^(s)  and  f_(di)^(c),

[0053] i=r,g,b. In the limit of vanishing anomalies we have:$\begin{matrix}{\left. f_{ci}^{s}\rightarrow h_{c}^{s} \right.,\left. f_{ci}^{c}\rightarrow h_{c}^{c} \right.,\left. f_{di}^{s}\rightarrow h_{d}^{s} \right.,\left. f_{di}^{c}\rightarrow h_{c}^{c} \right.,{i = r},g,{b.}} & (18)\end{matrix}$

[0054] It should be understood that other forms of ideal behavior can bespecified, and that the equations above, namely (9)-(12) are just oneexample representation.

[0055]FIG. 4 illustrates the main elements associated with theelectronic correction system 50, made accordance with the presentinvention. Electronic correction system 50 uses image processor 51 toeliminate optical anomalies common to image capture/display devices byintroducing an additional ‘inverse’ transformation in the processingchain discussed above. As before, electronic correction system 50includes optical elements (e.g. lenses, prisms, mirrors, etc.) andnon-optical elements (e.g. electronic sensors and panels) 52 and 54 someor all of which may introduce an optical anomaly into the imageprocessing chain as shown, as well as memory plane (i.e. storage plane)56 and 58 which is assumed not to introduce any anomalies. While both acapture and a display device are shown, it should be understood that thepresent invention could be easily adapted for use with a capture ordisplay device alone.

[0056] As discussed above in respect of FIG. 2, equations (2) to (7)describe the optical anomalies as transformations in the spatial andcolor domains. Image processor 51 applies arbitrary geometric (spatial)and color transformations to digital images. Spatial transformations arewell known and it should be understood that similar transformations canbe used to achieve color transformation. Implementation of these spatialtransformations can be implemented by image processor 51 using variouscommercially available image warp circuits (e.g. the s×W1 integratedcircuit manufactured by Silicon Optix of San Jose, Calif.) where certaincircuits work in the spatial domain and others can also apply colortransformations.

[0057] Image processor 51 processes each color from an imageindependently, allowing the transformations to be different for thethree channels. Image processor 51 is placed within the processing chainat the capture-display interface as shown. The transformations due toimage processor 51 will be denoted byf_(wr)^(s), f_(wg)^(s)  and  f_(wb)^(s)

[0058] (first subscript w) for the three spatial mappings, and ƒ_(wr)^(c),ƒ_(wg) ^(c) and ƒ_(wb) ^(c) for the three-color mappings. Theprocessing chain, as shown in FIG. 4 is now modified to: $\begin{matrix}\left. \left( {\overset{\rho}{x},O_{r},O_{g},O_{b}} \right)\rightarrow\left( {\overset{\rho}{x_{r}^{\prime}},\overset{\rho}{x_{g}^{\prime}},\overset{\rho}{x_{b}^{\prime}},O_{r}^{\prime},O_{g}^{\prime},O_{b}^{\prime}} \right) \right. & (19) \\{\overset{\rho}{x_{r}^{\prime}} = {f_{dr}^{s}\left( {f_{wr}^{s}\left( {f_{cr}^{s}\left( \overset{\rho}{x} \right)} \right)} \right)}} & (20) \\{\overset{\rho}{x_{g}^{\prime}} = {f_{dg}^{s}\left( {f_{wg}^{s}\left( {f_{cg}^{s}\left( \overset{\rho}{x} \right)} \right)} \right)}} & (21) \\{\overset{\rho}{x_{b}^{\prime}} = {f_{db}^{s}\left( {f_{wb}^{s}\left( {f_{cb}^{s}\left( \overset{\rho}{x} \right)} \right)} \right)}} & (22) \\{O_{r}^{\prime} = {f_{dr}^{c}\left( {f_{wr}^{c}\left( {f_{cr}^{c}\left( O_{r} \right)} \right)} \right)}} & (23) \\{O_{g}^{\prime} = {f_{dg}^{c}\left( {f_{wg}^{c}\left( {f_{cg}^{c}\left( O_{g} \right)} \right)} \right)}} & (24) \\{O_{b}^{\prime} = {{f_{db}^{c}\left( {f_{wb}^{c}\left( {f_{cb}^{c}\left( O_{b} \right)} \right)} \right)}.}} & (25)\end{matrix}$

[0059] To eliminate the optical anomalies, the transformations utilizedwithin image processor 51 are chosen such that equations (19) to (25)reduce to (13) to (17). Comparing equations (19) to (25) with equations(13) to (17) gives (it should be understood that multiplication in therelations below implies functional composition): $\begin{matrix}{f_{wr}^{s} = {f_{dr}^{s^{- 1}}h_{d}^{s}h_{c}^{s}f_{cr}^{s^{- 1}}}} & (26) \\{f_{wg}^{s} = {f_{dg}^{s^{- 1}}h_{d}^{s}h_{c}^{s}f_{cg}^{s^{- 1}}}} & (27) \\{f_{wb}^{s} = {f_{db}^{s^{- 1}}h_{d}^{s}h_{c}^{s}f_{cb}^{s^{- 1}}}} & (28) \\{f_{wr}^{c} = {f_{dr}^{c^{- 1}}h_{d}^{c}h_{c}^{c}f_{cr}^{c^{- 1}}}} & (29)\end{matrix}$

$\begin{matrix}{f_{wg}^{c} = {f_{dg}^{c^{- 1}}h_{d}^{c}h_{c}^{c}f_{cg}^{c^{- 1}}}} & (30) \\{f_{wb}^{c} = {f_{db}^{c^{- 1}}h_{d}^{c}h_{c}^{c}f_{cb}^{s^{- 1}}}} & (31)\end{matrix}$

[0060] The above equations are obtained by appropriately equating (19)to (25) with (13) to (17). For example, to derive (26), set${{f_{dr}^{s}\left( {f_{wr}^{s}\left( {f_{cr}^{s}\left( \overset{\rho}{x} \right)} \right)} \right)} = {h_{d}^{s}\left( {h_{c}^{s}\left( \overset{\varpi}{x} \right)} \right)}},$

[0061] which implies on the functional level thatf_(dr)^(s)f_(wr)^(s)f_(cr)^(s) = h_(d)^(s)h_(c)^(s).

[0062] Then multiplying both sides from the left by f_(dr)^(s⁻¹)

[0063] and from the right by f_(cr)^(s⁻¹),

[0064] and canceling the inverses, leads to (26). Other equations arederived similarly.

[0065] When image processor 51 applies the transformations in equations(26) to (31), the optical anomalies of distortion, color non-convergence(excluding axial chromatic aberration) and luminance (chrominance)non-uniformity will be eliminated.

[0066] Implicit in transformations (26) to (31) is the filtering thataccompanies any image warping. After the geometric transformationsf_(wr)^(s), f_(wg)^(s)

[0067] and f_(wb)^(s)

[0068] are applied; re-sampling (of all three channels) is needed togenerate the new image or pixel data. In image processor 51, thesefiltering operations often take place before the color transformationsf_(wr)^(c), f_(wg)^(c)  and  f_(wb)^(c)

[0069] are applied. The color mappings are then applied to the newfiltered pixel data, implying that the anomaly correctingtransformations (29) to (31) need to be based on the new image data.

[0070]FIG. 5 illustrates the specific processing sequence that occurswithin image processor 51, where a spatial filter 47 performs thefiltering operation denoted by F. The geometric data received for eachoutput pixel is used by spatial filter 47 to acquire a number of inputpixels for transform by FIR filter to an output pixel value. Also, again and offset stage 49 is utilized to ensure that uniformluminance/chrominance results as previously discussed. As a specificexample, transformation (29) will correct the luminance non-uniformityif the component mapping f_(cr)^(c⁻¹)

[0071] is chosen to act as follows: $\begin{matrix}{{{f_{cr}^{c^{- 1}}\left( {F\left( {f_{cr}^{c}\left( O_{r} \right)} \right)} \right)} = O_{r}},{{or}\quad {{equivalently}:}}} & (32) \\{{{h_{c}^{c}\left( {f_{cr}^{c^{- 1}}\left( {F\left( {f_{cr}^{c}\left( O_{r} \right)} \right)} \right)} \right)} = {h_{c}^{c}\left( O_{r} \right)}},} & (33)\end{matrix}$

[0072] or equivalently:

[0073] and so forth for the other cases. Here spatial filter 47 is usedto combine F with f_(cr)^(c⁻¹)

[0074] to eliminate the capture side luminosity anomaly. The filtering Fis implicit in all processing chains or equations that include an imagewarping circuit. Also implicit is the fact that the colortransformations are indirectly linked to the spatial transformations viathe filtering. This means that a change in the spatial transformationwill change the filtered pixel data, which in turn may require differentcolor transformations. The spatial mappings can also directly interactwith the color maps. It is contemplated that image processor 51 could beconfigured to allow for the dynamic modulation of the color mapparameters according to the spatial data (e.g. positional/derivativeinformation) being generated. This option is shown in FIG. 5 by thedotted arrow running between the space mapping element and thecolor-mapping element. The generation of the spatial mappings can beperformed by an attached processing block or offline (i.e. not withinimage processor 51), and stored as depicted by geometric transformdataset 57 and luminance transform dataset 59 in FIG. 5.

[0075]FIG. 6 illustrates another example electronic correction system100 which uses image processor 101 to implement a further generalizationof the transformations (26) to (31) to include an application specific“correction” or warping component (i.e. a component that is independentof the components that correct for the optical defects). As before,electronic correction system 100 includes optical elements (e.g. lenses,prisms, mirrors, etc.) and non-optical elements (e.g. electronic sensorsand display panels) 102 and 104 some or all of which may introduce anoptical anomaly into the image processing chain as shown, as well asmemory plane (i.e. storage plane) 106 and 108 which is assumed not tointroduce any anomalies.

[0076] Denoting the application specific mappings by ƒ_(a) ^(s) andƒ_(a) ^(c) (subscript a), equations (26) to (31) become: $\begin{matrix}{f_{wr}^{s} = {f_{dr}^{s^{- 1}}h_{d}^{s}f_{a}^{s}h_{c}^{s}f_{cr}^{s^{- 1}}}} & (34) \\{f_{wg}^{s} = {f_{d\quad g}^{s^{- 1}}h_{d}^{s}f_{a}^{s}h_{c}^{s}f_{cg}^{s^{- 1}}}} & (35) \\{f_{wb}^{s} = {f_{db}^{s^{- 1}}h_{d}^{s}f_{a}^{s}h_{c}^{s}f_{cb}^{s^{- 1}}}} & (36) \\{f_{wr}^{c} = {f_{dr}^{c^{- 1}}h_{d}^{c}f_{a}^{c}h_{c}^{c}f_{cr}^{c^{- 1}}}} & (37) \\{f_{wg}^{c} = {f_{d\quad g}^{c^{- 1}}h_{d}^{c}f_{a}^{c}h_{c}^{c}f_{cg}^{c^{- 1}}}} & (38) \\{f_{wb}^{c} = {f_{db}^{c^{- 1}}h_{d}^{c}f_{a}^{c}h_{c}^{c}{f_{cb}^{s^{- 1}}.}}} & (39)\end{matrix}$

[0077] With an application specific component, equation (33) becomes:$\begin{matrix}{{{f_{a}^{c}\left( {h_{c}^{c}\left( {f_{cr}^{c^{- 1}}\left( {F\left( {f_{cr}^{c}\left( O_{r} \right)} \right)} \right)} \right)} \right)} = {f_{a}^{c}\left( {\overset{\_}{F}\left( {h_{c}^{c}\left( O_{r} \right)} \right)} \right)}},} & (40)\end{matrix}$

[0078] where {overscore (F)}, an “effective filtering”, denotes thefiltering accompanying ƒ_(a) ^(s) in the absence of any anomalies. Now Fplays a role in both eliminating the anomalies and applying theapplication specific filtering due to ƒ_(a) ^(s). The filtering, F(and/or {overscore (F)}), is specified by correctly choosingf_(wi)^(s)  and  f_(wi)^(c),

[0079] i=r,g,b.

[0080] In general the mappings ƒ_(a) ^(s) and ƒ_(a) ^(c) can bedifferent for the three colors, however, for most practical purposesthese mappings are the same for all colors. It should be understood thathaving ƒ_(a) ^(s) be different for the three colors is equivalently to‘artificially’ introducing lateral chromatic aberration. It should benoted that ƒ_(a) ^(s) and ƒ_(a) ^(c) do not serve to correct the opticalanomalies, and their inputs/outputs are assumed to be anomaly free.Accordingly, the effective processing chain, with the anomalies removed,is shown in FIG. 6. Once image processor 101 has eliminated the opticalanomalies, image processor 101 effectively acts as an applicationspecific image warping circuit. FIG. 7 shows an effective processingchain that resides inside the image processor 101.

[0081] An example of electronic correction system 100 (i.e.capture/display device) is one that applies transformations (34) to (39)and that films (captures) a scene and projects (displays) it onto acurved screen. In this case ƒ_(a) ^(s) and ƒ_(a) ^(c) will correct forthe distortion (in position and/or color) that arises when projectingonto a curved screen. The remaining mapping components will correct forthe optical anomalies due to the capture/display optical/electroniccomponents. These remaining mapping components are device dependentmappings so that if the screen shape changes, only ƒ_(a) ^(s) and ƒ_(a)^(c) change, whereas the device dependent mappings would be unchanged.That is, mappingsf_(ci)^(s⁻¹), f_(di)^(s⁻¹), f_(ci)^(c⁻¹), f_(di)^(c⁻¹),

[0082] h_(c) ^(s), h_(d) ^(s), h_(c) ^(c) and h_(d) ^(c), need only becomputed once for a given device configuration. One application thatmight require different ƒ_(a) ^(c) mappings for each color is that ofcolor matching. In a situation where two overlapping images areprojected onto a screen, where the resulting image is required to becontinuous across the overlap region (i.e. where it is desired to splita large scene over several projectors), the three colors (in each image)in the overlap region need to be adjusted individually to present aseamless transition.

[0083] As previously noted, the method for representation andcomputation of the various mappings is an application/device dependent.The most general description of a mapping is in terms of a grid dataset.A grid dataset gives a representative set of pixels' input/outputpositions and colors values. Various methods can be used to generate agrid dataset. For optical elements, a ray-tracing program can be used todetermine input and output pixel positions. Test patterns can also berun through the system to determine color shifts. Data on input andoutput brightness levels can determine the grid dataset (in color space)that describes the luminance correcting map.

[0084] For application specific transforms (keystone, cylinder, sphere,etc), a mathematical model can be used to generate the datasetsanalytically. Accordingly, the grid datasets for the inverse functionsin transformations (34) to (39) can be computed by inverting the griddatasets from the ‘forward’ maps in transformations (2) to (7).

[0085] Once the individual datasets (corresponding to the five componentfunctions on the right of (34) to (39)) are determined, these can beconcatenated to give one grid dataset for each of the mappings (34) to(39). Concatenation can be done in several ways. One approach is tofirst fit or interpolate the individual datasets by 2D surfaces, such assplines, etc. These surfaces can then be functionally composed andevaluated in the correct order to obtain the final grid dataset. Thisprocess is repeated for each of the mappings (34) to (39). Thecomposition/evaluation operation simply uses the range of one surface asthe domain of the next surface, in order. Once we have the final sixgrid sets, these are then converted to a unifying 2D surface basedfunctional format for representing arbitrary transformations. There aresubstantial benefits of using this unifying functional format over a(pixel based) grid dataset format. The final result is that (34)-(39)can be expressed as: $\begin{matrix}{{f_{wi}^{s}\left( {x,y} \right)} = {\sum\limits_{m,n}\quad {a_{mn}^{i}x^{m}y^{n}}}} & \left( (41) \right. \\{{f_{wi}^{c}\left( {O_{i},x,y} \right)} = {{{\alpha_{i}\left( {x,y} \right)}O_{i}} + {\beta_{i}\left( {x,y} \right)}}} & \left( (42) \right. \\{{\alpha_{i}\left( {x,y} \right)} = {\sum\limits_{m,n}\quad {A_{mn}^{i}x^{m}y^{n}}}} & \left( (43) \right. \\{{\beta_{i}\left( {x,y} \right)} = {\sum\limits_{m,n}\quad {B_{mn}^{i}x^{m}y^{n}}}} & \left( (44) \right.\end{matrix}$

[0086] The conventional understanding relating to spatialtransformations, namely (41), can be applied to determine ((42) to((44). Instead of the pixel positions (as in (41)), the coefficients ofthe color map ((42) are being fitted in ((43) to ((44). The surfaces((43) to ((44) are determined in exactly the same manner as surface(41); only the meaning of the range is different. In ((42), the colormap is taken to be linear; higher order functions may be used though thephysical significance of higher order surfaces is not clear. Imageprocessor 51 (FIG. 5) and 101 takes ((42) to be linear in O_(i) and((43) to ((44) to be linear in x and y.

[0087]FIG. 8 illustrates another example electronic correction system150 which uses image processor 151 to achieve dynamic optical correctionand image warping. As shown the different transformations which definethe dynamic sequence can be computed using a map generator 162 andstored in an external memory 160. As before, electronic correctionsystem 150 includes optical elements (e.g. lenses, prisms, etc.) andnon-optical elements (e.g. electronic sensors and panels) 152 and 154some or all of which may introduce an optical anomaly into the imageprocessing chain as shown, as well as memory (i.e. storage plane) 156and 158 which is assumed not to introduce any anomalies.

[0088] Electronic correction system 150 uses image processor 151 tostore different mappings (i.e. more than one set of functions (41) to((44)). Using this approach, situations where the optical anomalies(and/or application specific warping) change with time (at a reasonablerate) can be addressed. This corresponds to adaptive anomaly correction,where the mappings adapt to the changing anomalies. The functions can bepre-computed according to the dynamic changes and then loaded as needed.Dynamic application specific effects can also be included, for example akeystone correction that changes with time. Although it has been assumedthat the mappings are computed offline, this need not be the case. Givenenough computational power, the mappings could be calculated online(i.e. inside an integrated circuit) which will eliminate the memoryblock required to store the various maps.

[0089] The functions discussed above (41) to ((44) can be stored inimage processor 51, 101 and 151 and in turn applied to the digital imagedata. The overall effect being that the ideal behavior is restored andonly application specific warping remains. This provides an effectiveelectronic solution to correcting optical anomalies, with the addedbenefit that any application specific image warping can also beincluded. Any changes to the application specific warping do not affectthe anomaly correction mappings.

[0090] The above electronic solution to optical anomaly correctiongreatly reduces the cost of a system. In particular, expensive opticalelements can be eliminated or replaced by cheaper components. Thecurrent circuit can correct distortion, lateral chromatic aberration andluminance/chrominance non-uniformity, as well as apply any applicationspecific image warping. A prime market for electronic correction system150, is in the area of high-resolution pixelated display systems whichare used in HDTV's, PC monitors, projection TV's etc.

[0091]FIG. 9 is a flowchart that illustrates the main process stepsassociated with the electronic correction method 200 of the presentinvention made accordance with the present invention. As discussed indetail above, electronic correction method 200 eliminates opticalanomalies common to image capture/display devices.

[0092] At step (202) an offline circuit, or possibly image processor 51,101 and 151 identifies various optical anomalies and collectivelyrepresents them as transformations of pixel data. The transformationscan be in spatial (positional) space and/or color space. As discussedabove, the most general way of specifying a transformation is throughgrid datasets, which give a representative set of pixels' input/outputpositions and color values. At step (204), the ideal behavior of theprocessing chain is also identified and represented as grid datasets.

[0093] At step (206), the anomaly correcting transformations (i.e. inthe form of grid datasets) is computed. These can be computed byinverting the grid datasets from the previous steps. At step (208), anyapplication specific transformations (grid datasets) are computed. Atstep (210), the anomaly correcting and application specifictransformations, or grid datasets, are concatenated into a singledataset. This gives one dataset, for each independent pixel variable,which specifies the complete ‘warp’ map that is to be appliedelectronically. There are six independent pixel variables, namely RGBpositional values and RGB color values.

[0094] At step (212), the concatenated grid datasets are converted to adevice specific functional format used by commercially available imagewarp circuits (e.g. the s×W1 integrated circuit manufactured by SiliconOptix of San Jose, Calif.). This functional description is capable ofrepresenting very general transformations in a manner that isappropriate for real-time, sub-pixel accurate, image processing.Finally, at step (214), the pixel data is digitally transformed,according to the pre-computed functional description, via the s×W1integrated circuit. As a result, the final image, having gone throughthe capture/warping/display processing chain, is free of any anomaliesand exhibits only the application specific image warping.

[0095] Electronic correction system 50, 100 and 150 and electroniccorrection method 200 provide efficient and cost-effective real-timeelectronic correction of aberrations that do not affect image sharpness,namely distortions and color non-convergence (excluding axial chromaticaberration) and luminance (or chrominance) non-uniformity. Each of theseeffects are modelled as transformations in either spatial (positional)space or color space. That is, once an aberration is expressed as apixel transformation (affecting either pixel positions or pixel colorcontent), it can be eliminated by applying the inverse transformation.The same approach also provides for application specific imageprocessing and dynamic anomaly correction. Electronic correction method200 is not restricted to display/projection devices or capture devices,but may be used in conjunction with any image processing chaincontaining a display and/or capture component. All corrections arerepresented as grid datasets or functions acting in spatial or colorspace, which allows different corrections to be concatenated throughfunctional compositions.

[0096] As will be apparent to those skilled in the art, variousmodifications and adaptations of the structure described above arepossible without departing from the present invention, the scope ofwhich is defined in the appended claims.

1. An electronic correction method for correcting a plurality of opticalanomalies associated with the capture and display of an optical imageprocessed through optical capture and display components having aparticular geometry, by compensation of the digital image pixel dataassociated with the optical image, said method comprising: (a)identifying and representing the optical anomalies associated with thephysical and geometrical characteristics of the capture and displayoptical components as an optical anomaly grid dataset; (b) identifyingand representing the ideal behavior of the capture and display opticalcomponents as an ideal grid dataset, (c) comparing the optical anomalygrid dataset with the ideal grid dataset and determining an anomalycorrecting transformation dataset by performing an inverse spatialtransform from the ideal grid dataset to the anomaly transform such thatfunctional composition of the anomaly correcting transformation with theoptical anomaly grid dataset reduces to the ideal grid dataset; (d)applying the anomaly correcting transformation dataset to the imagepixel data to produce corrected image pixel data which when viewed isfree of the optical anomaly.
 2. The method of claim 1, wherein saidoptical anomalies are selected from the group consisting of:distortions, color non-convergence, luminance non-uniformity, andchrominance non-uniformity.
 3. The method of claim 1, further comprisingcomputing an application specific grid dataset which represents imagescaling and geometric transformations selected from the group consistingof spherical, cylindrical, and keystone transformations, andconcatenating the optical anomaly grid dataset and the applicationspecific grid dataset wherein (d) consists of applying the concatenationof the optical anomaly grid dataset and the application specific griddataset to the pixel image data.
 4. The method of claim 1, wherein theanomaly correcting transformation dataset is converted to a devicespecific functional representation.
 5. The method of claim 1, furthercomprising storing a sequence of transformations to achieve dynamicanomaly correction.
 6. The method of claim 1, wherein the anomalycorrecting transformation dataset is defined in at least one of positionspace and color space
 7. The method of claim 1, wherein the anomalycorrecting transformation dataset defines a set of correcting geometrictransformations and a set of correcting color transformations.
 8. Themethod of claim 8, wherein (d) comprises (i) applying the set ofcorrecting geometric transformations; (ii) applying a set of filteringoperations; and (iii) applying the set of correcting colortransformations.
 9. The method of claim 1, wherein the pixel image datais associated with at least one of a capture and display device.
 10. Anelectronic correction system for correcting a plurality of opticalanomalies associated with the capture and display of an optical imageprocessed through optical capture and display components having aparticular geometry, by compensation of the digital image pixel dataassociated with the optical image, said system comprising an imageprocessor for: (a) identifying and representing the optical anomalies ofthe physical and geometrical characteristics of the capture and displayoptical components as an optical anomaly grid dataset; (b) identifyingand representing the ideal behavior of the image data processing chainas an ideal grid dataset; (c) comparing the optical anomaly grid datasetwith the ideal grid dataset and determining an anomaly correctingtransformation dataset by performing an inverse spatial transform fromthe ideal grid dataset to the anomaly transform such that functionalcomposition of the anomaly correcting transformation with the opticalanomaly grid dataset reduces to the ideal grid dataset; (d) applying theanomaly correcting transformation dataset to the image pixel data toproduce corrected image pixel data which when viewed is free of theoptical anomaly.
 11. The system of claim 10, wherein said opticalanomalies are selected from the group consisting of: distortions, colornon-convergence, luminance non-uniformity and chrominancenon-uniformity.
 12. The system of claim 10, further comprising anapplication module for providing an application specific grid datasetwhich represents selected image scaling and geometric transformationsselected from the group consisting of spherical, cylindrical, andkeystone transformations, said image processor being adapted toconcatenate the optical anomaly grid dataset and the applicationspecific grid dataset, said processor also being adapted to apply theconcatenation of the optical anomaly grid dataset and the applicationspecific grid dataset to the image pixel data.
 13. The system of claim10, wherein the image processor is adapted to convert the anomalycorrecting transformation dataset to a device specific functionalrepresentation.
 14. The system of claim 10, wherein the image processoris adapted to store a sequence of transformations to achieve dynamicanomaly correction.
 15. The system of claim 10, wherein the opticalanomaly grid dataset is defined in at least one of position space andcolor space.
 16. The system of claim 10, wherein the anomaly correctingtransformation dataset defines a set of correcting geometrictransformations and a set of correcting color transformations.
 17. Thesystem of claim 16, wherein the image processor is adapted to: (i) applythe set of correcting geometric transformations; (ii) apply a set offiltering operations; and (iii) apply the set of correcting colortransformations.
 18. The system of claim 10, further comprising an imagedisplay device coupled to the image processor for displaying saidcorrected image pixel data.
 19. The system of claim 10, furthercomprising an image capture device coupled to the image processor forcapturing said image pixel data.
 20. A computer-readable medium havingcomputer-readable code embodied therein for correcting a plurality ofoptical anomalies associated with the capture and display of an opticalimage processed through optical capture and display components having aparticular geometry, by compensation of the digital image pixel dataassociated with the optical image, by: (a) identifying and representingthe optical anomalies of the physical and geometrical characteristics ofthe capture and display optical components as an optical anomaly griddataset; (b) identifying and representing the ideal behavior of theimage data processing chain as an ideal grid dataset; (c) comparing theoptical anomaly grid dataset with the ideal grid dataset and determiningan anomaly correcting transformation dataset by performing an inversespatial transform from the ideal grid dataset to the anomaly transformsuch that functional composition of the anomaly correctingtransformation with the optical anomaly grid dataset reduces to theideal grid dataset; (d) applying the anomaly correcting transformationdataset to the image pixel data to produce corrected image pixel datawhich when viewed is free of the optical anomaly.
 21. Thecomputer-readable medium of claim 20, wherein said optical anomalies areselected from the group consisting of: distortions, colornon-convergence, luminance non-uniformity, and chrominancenon-uniformity.
 22. The computer-readable medium of claim 20, furthercomprising the step of computing an application specific grid datasetwhich represents selected image scaling and geometric transformationsselected from the group consisting of spherical, cylindrical, andkeystone transformations and concatenating the optical anomaly griddataset and the application specific grid dataset wherein (d) consistsof applying the concatenation of the optical anomaly grid dataset andthe application specific grid dataset to the pixel image data.
 23. Thecomputer-readable medium of claim 20, wherein the anomaly correctingtransformation dataset is converted to a device specific functionalrepresentation.
 24. The computer-readable medium of claim 20, furthercomprising storing a sequence of transformations to achieve dynamicanomaly correction.
 25. The computer-readable medium of claim 20,wherein the anomaly correcting transformation dataset is defined in atleast one of position space and color space
 26. The computer-readablemedium of claim 20, wherein the anomaly correcting transformationdataset defines a set of correcting geometric transformations and a setof correcting color transformations.
 27. The computer-readable medium ofclaim 26, wherein (d) comprises (i) applying the set of correctinggeometric transformations; (ii) applying a set of filtering operations;and (iii) applying the set of correcting color transformations.